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SECTION  I 
INTRODUCTION 


The  purpose  of  this  study  is  to  develop  and  implement  a model  to  aid 
in  the  choice  of  computational  systems  for  aircraft  simulator  crew 
training  devices.  This  requires  the  identification  and  organization  of 
computational  system  requirements  and  Air  Force  aims  and  philosophy. 
Once  these  are  recognized  and  characterized  in  terms  of  computer  attri- 
butes, a model  can  be  devised  to  judge  candidate  computer  systems. 
Attention  was  focused  on  those  attributes  which  define  tne  central 
processing  unit  of  a computer  and  those  items  which  determine  the  com- 
puter's ability  to  cost-effectively  handle  the  simulator  processing 
load  and  promote  commonality  in  the  Air  Force  inventory.  During  the 
course  of  this  study,  Singer  was  furnished  information  as  an  aid  in 
choosing  selection  criteria  and  identifying  Air  Force  aims,  interests 
and  philosophy.  The  information  was  in  the  form  of  specifications, 
contractor  submitted  proposals  for  specific  simulators,  guidelines  for 
evaluating  proposals  and  MIL-STD-876A.  Four  generalized  lists  of 
variables  prepared  by  Air  Force  Logistics  Command  (AFLC)  and  a copy 
of  a conceptual  plan  formulating  incentives  for  reducing  operating 
and  support  costs  were  also  provided  and  were  considered  in  preparing 
the  "costs"  portion  of  the  model.  Information  about  simulator  use 
was  collected  from  questionnaires  sent  to  numerous  Air  Force  simu- 
lator sites.  Personal  visits  to  simulator  bases  were  made  by  Singer 
personnel  to  obtain  insight  and  information  not  easily  covered  by  a 
questionnaire  and  to  allow  Air  Force  personnel  to  volunteer  ideas 
which  may  not  have  occurred  to  the  investigators.  Information  from 
these  many  sources  was  pooled  with  the  personal  experience  of  those 
directly  involved  in  the  study.  The  result  was  a model  formulated  to 
rank  candidate  computers  in  their  abilities  to  perform  simulator 
processing  tasks.  The  model  was  implemented  in  the  form  of  a computer 
program  named  "CRITIC."  "CRITIC"  is  an  acronym  for  Candidate  Rank- 
ing and  Inventory  Traits  Inspection  for  Commonality. 
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SECTION  II 


REVIEW  OF  COMPUTATION  SYSTEM  SELECTION  PRACTICES 

MIL-STD-876A  (USAF),  "Military  Standard  — Digital  Computation  Systems 
for  Real-Time  Training  Simulators"  dated  8 July  1971,  . .covers 

the  general  characteristics  and  configurations  of  digital  computation 
systems  used  in  (USAF)  real-time  training  simulators  and  the  general 
guidelines  for  mathematical  models."  This  standard,  which  supersedes 
MIL-STD-876  (USAF)  of  21  February  1967,  is  ".  . .mandatory  for  use 
by  all  activities  under  the  cognizance  of  the  Air  Force  effective  as 
of  date  of  issue."  In  reviewing  these  standards,  we  discerned  a 
need  for  flexibility  in  computer  system  selection  criteria  specifica- 
tion and  application  in  our  model.  Obviously,  the  model  should 
recognize  "requirements"  as  one  category  of  selection  criteria.  Also, 
some  allowances  should  be  made  for  "preferences"  which  may  be  satis- 
fied in  varying  degrees  as  opposed  to  the  more  limited  interpretation 
of  "requirements."  "Preference"  criteria  can  be  subdivided  or  asso- 
ciated with  differing  incremental  values  (weights)  with  respect  to 
"commonality,"  "low  risk,"  and  "support  objectives."  These,  together 
with  "requirements,"  "performance,"  and  life  cycle  cost  evaluations, 
are  considered  to  comprise  an  adequate  set  of  categories  of  criteria 
to  be  included  in  the  model. 

Excerpts  relating  to  digital  computation  systems  from  specifications 
for  C-141A,  T-37  and  T-38,  A-7D,  F-15A,  and  CH-3E  and  HH-53C  air- 
craft simulators  were  reviewed.  The  trend  has  been  toward  more 
explicit  statements  in  terms  of  performance  requirements  and  partic- 
ular main  frame  computer  systems  characteristics.  Also  require- 
ments specified  for  spare  time,  spare  memory,  and  spare  input/output 
(I/O)  channels,  etc.,  have  increased  from  10%,  10%,  and  10%  to  25%, 
25%,  and  15%  respectively.  In  addition,  the  more  recent  specifica- 
tions have  been  more  explicit  than  MIL-STD-876A  in  the  manner  in 


which  the  spare  time  requirement  shall  be  met.  We  decided  the 
model  should  accept  spare  time  and  memory  requirements  as  input 
parameters  for  candidate  computers'  merit  assessment  and  to  apply 
the  MIL-STD-876A  parameters  independently. 

Excerpts  from  successful  bidders'  proposals  for  the  T-37  and  T-38, 
A-7D,  F-15A,  CH-3E  and  HH-53C  aircraft  simulators  were  also  reviewed. 
In  each  case,  the  computer  system  selected  employed  one  or  two 
general  purpose  digital  computers  having  a 24-bit  word  length.  With 
one  exception,  the  computers  selected  were  different  models  built 
by  the  same  computer  manufacturer,  Datacraft,  now  a subsidiary  of 
the  Harris  Corporation.  The  exception  was  a dual  Honeywell  DDP-324 
configuration  for  which  a single  Datacraft  6024/1  configuration  was 
substituted  during  the  A-7D  simulator  development.  A variety  of 
other  machines  were  considered  as  candidates,  among  which  16,  18, 

24,  and  32  bit  word  lengths  were  represented. 

Computer  system  hardware  cost  (predominantly  central  processor, 
processor  options,  and  core  memory)  was  a primary  factor  in  select- 
ing main-frame  computers  (or  eliminating  others  from  further  con- 
sideration) from  those  candidates  which  were  capable  of  being  con- 
figured to  handle  the  estimated  processing  load  and  with  the 
specified  spare  capacity  requirements.  Other  factors  affecting 
development  and  life  cycle  costs  were  considered  in  varying  degrees. 
Total  costs, of  some  items  such  as  computer  system  hardware  increase 
with  the  number  of  identical  systems  procured.  Costs  of  other  items, 
such  as  software  programming  (both  for  initial  development  and  for 
life-cycle  modifications),  do  not.  Cost  factors  as  quantified  in 
relative  terms  in  the  various  technical  proposal  excerpts  are  an 
insufficient  basis  for  assessing  candidates  in  general.  In  future 
procurements  additional  cost-related  factorr  may  reasonably  be 
expected  to  influence  the  selection.  For  this  reason,  provision  was 
made  in  the  model  to  allow  a wider  variety  of  cost  estimates  to  be 
included. 
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Analyses  relating  to  application  of  computer  system  performance  criteria, 
were  apparently  performed  in  essentially  the  same  manner,  although 
some  of  the  relevant  details  were  not  always  presented  within  the 
references  furnished  for  review.  The  general  approach  consisted  of 
defining  a number  of  functional  "modules"  which  collectively  charac- 
terized the  real-time  simulation  computational  load  in  terms  of  in- 
structions per  second  and  storage  requirements  for  programs  and  data. 

The  instructions  per  second  figures  were  derived  from  products  of 
module  execution  frequency  and  number  of  instructions  executed  within 
the  module  (including  internal  loops)  representing  some  heavily 
loaded  or  worst  case  situations.  Differing  conventions  may  have  been 
employed  with  respect  to  instructions,  constants,  parameters  and 

i 

variables  of  various  types  in  storage  requirements  estimation.  How- 
ever, estimated  storage  requirements  were  generally  expressed  in 
words  with  some  appropriate  accounting  for  partial  word  and/or  multiple 
word  data  elements  and/or  instructions.  It  was  decided  to  permit 
computational  load  estimates  to  be  input  to  the  model  in  modular 
fashion  so  that  the  functional  breakdown,  if  desired,  could  be  included 
in  the  computer  listing  of  the  model  input.  This  would  still  enable 
total  loading  estimates  to  be  input  as  fictitious  "modules." 

Computational  capability  of  the  various  candidates  were  generally 
represented  in  terms  of  instructions/second  capacity  of  one  or  multiple 
processors  with  some  appropriate  accounting  for  memory  access  delays 
due  to  input /output,  indexed  or  indirect  memory  reference,  etc.  In- 
structions/second capacity  for  a given  processor  was  inversely  related 
to  a weighted  average  instruction  execution  time.  In  essence,  a set 
of  representative  instructions,  characterizing  the  overwhelming 
majority  of  actual  instructions  used  in  a simulation  problem,  were 
each  assigned  a probability  of  occurrence  in  a simulation  problem 
based  upon  data  from  previous  presumably  similar  simulations.  These 
weight  factors  are  generally  referred  to  as  "usage"  factors  and  the 
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set  is  generally  referred  to  as  a "simulation  mix."  The  effective 
time  for  a given  processor  to  execute  each  representative  instruction 
is  multiplied  by  the  corresponding  instruction  mix  usage  factor. 

The  products  are  then  summed  to  obtain  the  weighted  average  instruc- 
tion execution  time.  We  extended  this  approach  in  our  model,  by 
permitting  several  instruction  mix  sets  to  be  used  in  characterizing 
a collection  of  software  modules.  This  enables  software  modules 
which  may  have  significantly  different  characteristics  to  be  associated 


SECTION  III 

DATA  FROM  USAF  SIMULATOR  INSTALLATIONS 

The  Statement  of  Work  under  which  this  study  was  performed  stipuiaLci 
that  data  be  gathered  from  simulator  facilities  at  Plattsburgh, 

Myrtle  Beach,  Altus,  and  Rill  Air  Force  Bases.  To  facilitate  this 
data  gathering,  we  mailed  sets  of  data  collection  forms,  "Question- 
naires," to  cognizant  USAF  personnel  at  the  respective  simulator 
facilities.  These  seven  page  questionnaires  incorporated  a number 
of  contributions  from  the  USAF  study  team.  The  completed  question- 
naires provided  a convenient  basis  for  further  discussions  during 
our  visits  at  the  bases.  Early  in  the  study  it  was  decided  that  the 
data  base  might  be  significantly  enhanced  if  questionnaire  responses 
from  additional  USAF  simulator  installations  were  also  obtained, 
even  though  follow-up  visits  could  not  be  made  within  the  scope  of 
the  study  contract.  Accordingly,  cooperation  was  requested  by  the 
USAF  study  manager  and  questionnaire  responses  were  subsequently 
obtained  from  ten  additional  USAF  bases.  As  responses  were  received, 
a copy  of  each  was  forwarded  to  the  USAF  study  manager.  A copy  was 
also  forwarded  to  the  corresponding  simulator  contractor  so  their 
cognizant  people  might  have  the  opportunity  to  amplify  or  clarify 
the  information  therein.  Annotated  questionnaire  responses  are  pre- 
sented in  the  Data  Supplement  to  the  Final  Report.  Annotations 
reflect  contractor  review,  follow-up  visits,  or  both.  Additional 
information  obtained  during  the  follow-up  visits  (e.g.,  organization 
charts)  is  included  with  the  corresponding  questionnaire  responses 
in  the  Data  Supplement. 

Simulator  installations  at  Plattsburgh,  Myrtle  Beach,  Altus,  and 
Hill  Air  Force  bases  were  visited  during  the  period  May  12-23,  1975. 
Personnel  at  each  facility  were  cooperative  even  though  these  visits 
were  imposing  additional  demands  on  their  time.  Approximately  a day 
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and  a half  was  spent  at  each  base  observing  simulator  and  computer 
operations,  reviewing  questionnaire  responses,  and  gathering  addi- 
tional information  relevant  to  hardware  and  software  maintenance 
and  support.  Much  of  the  information  was  received  from  informal 
discussions  with  USAF  military  and  civilian  personnel.  Our  atten- 
tion was  focused  on  growth  history,  maintenance  and  support  of  the 
computational  systems,  with  primary  emphasis  on  hardware  and  soft- 
ware employed  in  the  main  computer (s).  Additional  information  was 
sought  concerning  the  overall  simulators  so  the  findings  would  be 
presented  in  proper  perspective. 

Two  FB-111A  mission  simulators  are  located  at  Plattsburgh  AFB;  a 
third  is  located  at  Pease  AFB.  Each  of  these  includes  one  replica 
of  the  FB-111A  cockpit  containing  pilot  and  navigator  stations  on 
a five  degrees-of-f reedom  motion  base.  Each  mission  simulator 
incorporates  three  Xerox  Data  Systems  (XDS)  Sigma  5 central  process- 
ing units  (CPU's)  in  a multiprocessor  configuration  with  a total  of 
132K  32-bit  words  of  core  memory. (NOTE:  K=2^®  « 1024  is  used  con- 

sistently in  this  summary.)  CPU's  A,  B and  C share  16K  words  via 
a six-way  access  port  and  have  private  memories  of  40K,  44K  and  32K 
words,  respectively.  A Raytheon  703  minicomputer  and  a Singer 
digital  target  generator  (DTG)  are  employed  as  auxiliary  processors 
in  conjunction  with  the  tactics  simulation.  The  FB-lilA  Bomb/Nav 
simulator  is  less  complex;  in  essence,  most  pilot  instruments  and 
controls  are  inactive  and  the  cockpit  replica  is  situated  on  a fixed 
base.  The  computation  system  includes  two  XDS  Sigma  5 CPU's  with  a 
total  of  76K  32-bit  words  of  core  memory,  a Raytheon  703  and  a DTG. 

Simulator  training  costs  associated  with  primary  personnel,  operation 
and  maintenance  and  utilities  were  stated  to  be  in  the  neighborhood 
of  $165/hr.  Approximately  sixty  USAF  personnel  provide  on-site 
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maintenance  and  operations  support  for  the  three  simulators  at 
Plattsburgh.  Of  these,  twelve  have  had  a year  or  more  experience 
with  this  equipment.  (The  FB-111A  mission  simulators  have  been 
on-site  for  nearly  5 years.)  Approximately  680  hours  of  specialized 
on-site  training  (70  - 75%  classroom,  25  - 30%  on-the-job)  is  given 
new  recruits  after  completion  of  the  appropriate  Air  Training  Command 
(ATC)  technical  training  course  for  simulators  (i.e.,  approximately 
32  weeks  of  342  "Flights"  or  35  weeks  of  343  "Tactics"  schools). 
Significant  amounts  of  on-site  training  is  also  required  by  personnel 
who  have  been  reassigned  from  other  USAF  simulator  installations 
where  the  computation  equipment  is  substantially  different.  Much 
of  the  on-site  training  is  conducted  or  aided  by  AFETS  personnel. 

Most  of  the  troubleshooting  and  other  unscheduled  maintenance  is 
accomplished  or  guided  by  the  on-site  field  service  representatives. 

One  A-7D  simulator  is  located  at  Myrtle  Beach  AFB.  Four  duplicate 
simulators  are  located  at  Davis-Monthan  AFB  (Arizona) , England  AFB 
(Louisiana) , Rickenbacker  AFB  (Ohio) , and  Buckley  AFB  (Colorado) . 

Each  of  these  includes  one  replica  of  the  single-seat  A-7D  cockpit 
on  a four  degrees-of-freedom  motion  base.  One  Datacraft  6024/1  CPU 
with  48K  24-bit  words  of  core  memory  is  incorporated  in  each  simulator. 
The  Myrtle  Beach  computer  complex  includes  a keypunch,  card  reader, 
line  printer,  and  disc  drive  in  addition  to  the  peripherals  (paper 
tape  reader,  paper  tape  punch,  ASR-35  and  KSR-35  teletypewriters, 
and  disc  drive)  included  in  the  other  complexes. 

On-site  maintenance  and  simulator  operations  support  is  accomplished 
by  15  USAF  personnel  (including  three  administrators)  and  one  field 
service  representative.  It  was  mentioned  that  a Civilian  Technical 
Assistant  (CTA)  was  being  sought  to  augment  the  staff  for  continuity 
in  maintenance  training.  Approximately  one  year  of  classroom  and 
hands-on  training  (after  36  weeks  of  technical  school)  is  required 


8 


to  qualify  a maintenance  technician  for  independent  work  on  this 
system.  About  2.5  months  of  this  is  devoted  to  computer  training. 
Assembly  and  machine  language  familiarization  is  included  in  the 
course  requirement;  compiler  familarization  is  made  available  as 
an  option.  A three-man  Development  Technician  Team  (DTT)  presently 
at  the  Myrtle  Beach  A-7D  simulator  facility  provides  key  organic 
support  for  this  and  the  other  A-7D  simulators  used  by  TAC.  Their 
role  in  simulator  hardware  and  software  change  implementation  and 
in  related  development  efforts  varies  depending  upon  the  nature  of 
the  change.  Arrangements  have  been  made  with  the  computer  vendor 
(Harris  Corporation,  which  assimilated  Datacraft)  for  on-call  service 
when  special  computer  system  repairs  are  required.  This  has  been 
exercised  several  times  since  installation  of  the  first  three  simu- 
lators. 

I 

Three  C-141A  simulators  are  located  at  Altus  AFB.  Five  additional 
C-141A  simulators  are  operational  at  other  bases.  Each  includes 
one  replica  of  the  C-141A  cockpit  with  pilot,  co-pilot  and  flight 
engineer  stations  and  two  onboard  instructor  stations  on  a four 
degrees-of-freedom  motion  base.  Six  of  the  eight  C-141A  simulators 
each  incorporate  one  Systems  Engineerings  Laboratories  (SEL)  840  MC 
computer  with  44K  24-bit  words  of  core  memory.  Two  of  these  simu- 
lators are  at  Altus.  Two  of  the  eight  C-141A  simulators  were  supplied 
with  two  Control  Data  Corporation  (CDC)  924  computers  in  each.  Each 
of  these  systems  incorporates  a total  of  48K  24-bit  words  of  core 
memory  including  16K  words  added  since  installation.  One  of  these 
simulators  is  at  Altus.  Two  C-5A  simulators  are  located  at  Altus 
AFB.  Three  additional  C-5A  simulators  are  located  at  Dover  AFB 
(Delaware)  and  Travis  AFB  (California) . Each  includes  one  replica 
of  the  C-5A  cockpit  with  pilot,  co-pilot,  flight  engineer,  and 
navigator  stations  and  three  onboard  instructor  stations  on  a four 
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degrees-of-freedom  motion  base.  These  simulators  each  incorporate 
one  SEL  840A  computer  and  one  SEL  840MP  computer  with  32K  and  40K 
words  of  24-bit  core  memory,  respectively,  including  8K  words  added 
since  initial  installation, 

Cost  per  hour  of  training  time  in  the  C-141A  and  C-5A  simulators  was 
stated  to  be  $92  and  $126,  respectively,  excluding  the  use  of  visual 
systems.  Cost  per  hour  of  visual  systems  (4  cockpit  displays  plus 
2 camera-models,  etc.)  usage  was  stated  to  be  $80.  Cost  per  hour  of 
training  in  a C-5A  aircraft  was  stated  to  be  in  the  neighborhood  of 
$3600.  Three-shift  operation  and  maintenance  support  for  the  five 
flight  simulators  at  Altus  AFB  is  provided  by  45  personnel.  In 
addition  nine  people  are  assigned  for  visual  system  maintenance. 

Four  people  are  assigned  to  a maintenance  and  engineering  prototyping 
group  "Mod  Squad"  whose  function  is  similar  to  that  of  the  DTT  for 
TAC  simulators.  Maintenance  and  operations  technicians  receive 
technical  school  (i.e.,  basic  f light/tactics  simulator  school) 
follow-up  training  in  a program  which  alternates  classroom  work  and 
hands-on  experience  in  sessions  several  weeks  in  length.  The  course 
includes  the  academic  portions  of  the  ground  school  training  received 
by  aircrew  personnel.  A list  of  the  in-house  training  courses  and 
the  number  of  hours  associated  with  each  precedes  the  questionnaire 
responses  from  Altus  in  the  Data  Supplement.  With  the  exception  of 
"Mod  Squad"  prototype  work,  the  overwhelming  majority  of  simulator 
maintenance  effort  at  Altus  was  stated  to  be  devoted  to  equipment 
other  than  the  digital  computers. 

It  was  suggested  that  overall  life  cycle  costs  might  be  reduced  if 
specifications  incorporated  more  stringent  I/O  system  design  require- 
ments. Prohibiting  power  supplies  to  be  loaded  above  75/5  or  even 
50%  of  rated  capacity  in  order  to  prolong  their  useful  life  was  an 
example  specifically  proposed.  It  was  suggested  that  a detailed 


study  of  the  I/O  interface  systems  (convertors,  multiplexors,  relays, 
cables,  connectors,  etc.)  might  be  more  beneficial  to  the  USAF  than 
the  present  study.  Such  a study  would  presumably  investigate  pres- 
ently available  equipment  and  system  design  alternatives  (e.g., 
individual  vs.  multiplexed  A/D  and  D/A  convertors)  and  recommend 
specification  standards  aimed  at  overall  life  cycle  cost  reduction. 
Additional  relevant  suggestions  are  included  in  the  questionnaire 
response  from  McGuire  AFB  in  the  Data  Supplement.  It  was  noted  during 
the  discussions  that  the  autopilot  and  fire  suppression  systems  were 
the  only  features  which  had  been  incorporated  in  the  simulators  prior 
to  their  installation  or  use  in  the  actual  aircraft.  The  desire  was 
expressed  to  have  all  simulator  changes  precede  or  at  least  parallel 
aircraft  system  changes  for  maximum  training  benefit.  We  were  in- 
formed that  the  need  to  update  radio  station  data  in  the  simulators 
was  usually  discerned  by  aircrew  personnel  who  noticed  differences 
between  simulated  and  real-world  station  parameters.  Altus  personnel 
suggested  that  this  situation  could  be  improved  upon  considerably  if 
a standard  library  of  radio  station  parameters  were  maintained  in 
computer  compatible  format  and/or  if  periodic  change  notices  were 
made  available  to  using  installations  in  a standard  format  which 
could  be  converted  into  whatever  specific  internal  formats  are  required. 
Thus,  for  example,  updating  of  radio  aids  data  for  each  of  the  C-141A 
and  C-5A  simulators  could  be  done  at  Altus  AFB  in  a systematic  and 
highly  mechanized  manner. 

The  CH-3E  and  HH-53C  simulators  at  Hill  AFB  are  one  of  a kind  train- 
ing devices.  Each  includes  a replica  of  the  respective  helicopter 
cockpit  on  a six  degrees-of-freedom  motion  base.  The  CH-3E  includes 
pilot,  co-pilot,  and  instructor  stations  and  the  larger  HH-53C  also 
includes  a flight  mechanic  station.  Instructor  stations  are  of  the 
keyboard-CRT  type.  These  simulators  each  incorporate  one  Datacraft 
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6024/3  CPU  with  40K  24-bit  words  of  core  memory  with  an  ASR-35  tele- 
typewriter and  a moving  head  disc.  A card  reader  and  a line  printer 
are  also  attached  to  the  CH-3E's  computer.  We  were  favorably  impressed 
by  the  compactness  and  flexibility  provided  by  the  keyboard-CRT  instruc- 
tor station.  We  were  told,  however,  that  the  instructors  consider  the 
necessity  of  altering  some  simulation  variables  such  as  wind  speed  and 
direction  via  the  keyboard  to  be  a serious  disadvantage. 

Simulator  operation  costs  approximately  $29/hour  including  mainte- 
nance, supplies,  utilities,  etc.,  and  excluding  instructor  and  trainee 
costs.  Comparable  cost  for  helicopter  operation  was  stated  to  be 
$557/hour.  All-inclusive  costs  for  simulator  training  were  stated  to 
be  approximately  $107/hour.  An  organization  chart  showing  authorized 
and  assigned  personnel  categories  for  administration,  aircrew  instruc- 
tion, and  maintenance  at  the  helicopter  simulator  facility  is  included 
with  the  questionnaire  responses  in  the  Data  Supplement.  Six  USAF 
personnel  are  assigned  to  the  maintenance  section. 

At  the  various  simulator  facilities  sampled,  software  modifications 
were  generally  associated  with  aircraft  configuration  changes,  gaming 
area  data  changes  (e.g.,  addition  or  deletion  of  radio  stations  and/ 
or  alteration  of  station  parameters) , correction  if  discrepancies 
between  simulator  and  aircraft  systems  and  additions  to  enhance  train- 
ing (e.g.,  introduce  new  malfunctions  for  instructor  selection)  or 
increase  operational  convenience  (e.g.,  add  a pre-programmed  mission). 
USAF  personnel  at  most  of  the  installations  reported  an  average 
program  modification  frequency  of  once  per  month.  A lower  figure, 
three  to  four  modifications  per  year,  was  reported  for  the  F-4E 
simulator  at  Chanute  AFB  and  flights  modules  were  stated  to  be  those 
most  often  modified.  In  contrast,  a monthly  average  modification 
frequency  was  reported  for  the  F-4E  simulator  at  Eglin  AFB  and  tactics 
modules  therein  were  stated  to  be  those  most  often  modified.  Both 
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sites  reported  the  modifications  involved  both  hardware  and  software 
changes.  The  Eglin  AFB  response  attributed  the  modifications  to  air- 
craft configuration  changes.  The  discrepancy  in  reports  for  presumably 
identical  simulators  may  reflect  substantially  different  time  periods 
spanned  by  the  "averages."  Weekly  software  modifications  were  reported 
to  occur  in  the  F-111D  simulator  at  Cannon  AFB  and  in  the  F-111F  sim- 
ulator at  Mountain  Home  AFB.  A tabulation  indicating  a total  of  1466 
cumulative  software  patches  associated  with  various  simulation  routines 
in  the  FB-111A  mission  simulator  was  obtained  at  Plattsburgh  AFB 
approximately  55  months  after  installation.  The  corresponding  average 
is  26.7  patches  per  month.  The  majority  of  these  1466  patches  were 
associated  with  navigation  (21.3%),  countermeasures  (21.2%),  bombing 
(17.5%)  and  flight  (12.8%);  twelve  additional  categories  of  functional 
routines  each  had  6.2%  or  fewer  patches  (27.2%  of  total).  Software  in 
the  Advanced  Simulator  for  Undergraduate  Pilot  Training  (ASUPT)  is 
subject  to  "continuous"  modification  according  to  the  questionnaire 
response  from  Williams  AFB  which  reported  modification  efforts  con- 
centrated on  motion,  flight,  and  g-seat  simulation  ranging  from 
"minor  changes  in  the  data  base  or  some  k-factor  to  complete  redesign 
of  an  entire  module."  The  ASUPT  is  being  employed  as  a research  tool 
by  Air  Force  Human  Resources  Laboratory  (AFHRL)  as  originally  intended; 
it  is  the  "youngest"  simulator  included  in  the  survey.  Software  in 
the  Simulator  for  Electronic  Warfare  Training  (SEWT)  at  Mather  AFB  is 
reportedly  modified  "to  meet  changes  of  training  criteria  and  improve 
training  equality"  (no  frequency  given)  and  modules  dealing  with 
printed  output,  student  evaluation,  and  equipment  displays  are  those 
most  often  modified. 
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SECTION  IV 

MODEL  AND  PROGRAM  DEVELOPMENT 

Computer  selection  criteria  were  organized  into  a model  which  was 
implemented  in  the  FORTRAN  IV  language.  The  program  was  tested  on 
an  in-house  computer.  Several  source  card  changes  enabled  the 
program  to  be  demonstrated  and  installed  at  the  CDC-6600  facility  in 
Building  676,  Area  B,  at  Wright-Patterson  AFB. 

The  CRITIC  (Candidate  Ranking  and  Inventory  Traits  Inspection  for 
Commonality)  computer  program  provides: 

1.  Computation  assistance  in  estimating  computer-related  costs  for 
a specified  simulator  procurement.  Separate  CRITIC  runs  can  compare 
candidates  on  the  basis  of  cost  projects  made  for  prototype  develop- 
ment, procurement  cycle,  or  for  the  anticipated  life  cycle  of  the 
simulator; 

2.  Computation  assistance  in  estimating  computer  loading  require- 
ments, and  assessment  of  candidate  configurations'  responsiveness 

in  terms  of  spare  time  and  memory  provisions  (including  multicockpit, 
multiprocessor  and  multicomputer  configurations); 

3.  A systematic  and  flexible  means  of  reviewing  computer  systems  in 
the  USAF  inventory  with  respect  to  meeting  required  characteristics 
(i.e.,  identifying  potential  candidates)  and  assessing  commonality 
among  systems  in  inventory; 

4.  A systematic  and  flexible  means  of  assessing  each  candidate 
system's  merit  from  the  standpoints  of  possessing  required  charac- 
teristics, commonality  with  USAF  inventory,  low  risk,  and  potential 
for  meeting  USAF  support  objectives  over  the  simulator  life  cycle. 
These  and  cost-performance  merit  are  combined  in  an  overall  merit 
rating  for  each  candidate; 

5.  Ranking  of  candidate  computer  systems  in  each  merit  category. 
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SECTION  V 
CONCLUSIONS 


It  is  concluded  that  the  CRITIC  model  and  computer  program  can  be 
a useful  tool  to  aid  in  the  selection  of  computers  for  use  in  USAF 
simulators.  One  important  aspect  of  its  usefulness  is  the  built-in 
flexibility  which  enables  the  program  to  adapt  to  changing  numbers 
of  criteria  in  various  categories  and  values  of  the  associated  weight- 
ing factors,  and  even  the  nature  of  characteristics  of  inventory  and 
candidates.  The  basic  structure  of  the  attribute  tables  and  the 
method  of  characteristics  testing  is  such  that  the  qualitative  defi- 
nitions of  most  of  the  listed  values  can  be  altered  to  suit  the 
needs  of  future  evaluations  (and  the  corresponding  tabular  data 
quantifying  same  can  be  altered  accordingly)  in  a manner  transparent 
to  the  program.  Also,  since  the  program  is  written  entirely  in 
FORTRAN  IV  and  all  input  data  (both  DATA  BASE  TAPE  and  RUN  INPUT  DECK) 
are  in  80-column  Hollerith  card  image  form,  the  program  and  its  data 
base  media  are  readily  adaptable  for  use  in  a wide  variety  of  computer 
installations.  (Minor  modifications  enable  the  CRITIC  program  to  be 
run  in  a machine  which  has  at  least  24K  core  words  of  24-bits  or 
greater  and  whose  FORTRAN  IV  compiler  supports  DECODE  statements.) 

It  should  be  emphasized  that  the  CRITIC  program  is  a tool  to  aid  in 
selection  of  computers  and  not  a computer  selector  or  a contractor 
selector.  As  such  it  can  reasonably  be  expected  to  focus  attention 
on  apparent  strong  points  or  weak  points  of  various  candidate  con- 
figurations in  a uniform  manner.  Like  other  computer  programs,  it 
is  susceptible  to  the  GIGO  (garbage-in,  garbage  out)  syndrome. 

Careful  and  objective  scrutiny  of  all  candidate  configuration  charac- 
terizations (likewise,  inventory  computer  characterizations),  as 
well  as  the  candidate-dependent  cost-related  inputs  should  maximize 
the  validity  of  the  candidate  rankings. 
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SECTION  VI 
RECOMMENDATIONS 

Complete  characterization  of  the  digital  computers  in  the  USAF 
simulator  inventory  was  beyond  the  scope  of  the  present  study.  We 
have  supplied  a card  deck  which  served  as  the  primary  medium  from 
which  a "DATA  BASE  TAPE"  was  created  for  CRITIC  program  verification. 

A listing  of  the  deck  is  included  in  the  CRITIC  program  documentation. 

A comprehensive  DATA  BASE  TAPE  may  be  prepared  following  this  example. 
The  necessary  data  for  preparing  these  representations  (attributes 
data  tables)  may  be  accumulated  from  a variety  of  sources  including 
computer  vendor  publications,  simulator  procurement  and  verification 
records,  GSA  catalogs,  AFLC  inventory  data,  and  simulator  vendors. 

We  recommend  that  the  USAF  take  appropriate  action  to  initiate  prepara- 
tion of  a comprehensive  DATA  BASE  TAPE  and  to  institute  procedures 
for  its  periodic  review  and  updating  (to  reflect  inventory  changes, 
revised  computer  system  specifications,  etc.)  consistent  with  antici- 
pated usage  of  the  CRITIC  program. 


i 

i- 


Before  encoding  additional  computer  system  descriptions  in  conjunction 
with  preparation  of  a comprehensive  DATA  BASE  TAPE,  consideration 
should  be  given  to  augmenting  the  computer  attributes  definition  list 
(i.e.,  assigning  some  additional  definitions  to  various  "spare"  table 
entries)  and  rearranging  some  of  the  items  in  the  table.  Time  con- 
straints for  checkout  and  delivery  of  the  program  precluded  our  doing 
a second  iteration  on  the  attributes  table  layout.  A review  of  this 
area  is  desirable  to  enhance  the  information  content  of  the  charac- 
terizations and  to  effect  more  optimum  grouping  of  table  entries. 

The  latter  should  facilitate  preparation  and  checking  of  the  associated 
data  for  both  inventory  and  candidate  computer  systems. 

There  are  some  areas  related  to  this  study  but  beyond  its  scope  which 
we  believe  merit  further  investigation.  They  are: 
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1.  How  effectively  can  higher  level  languages  such  as  FORTRAN  be 
used  in  real-time  simulator  programs?  There  are  some  obvious  benefits 
from  relative  ease  of  program  development  and  maintenance.  There  are 
also  penalties  for  high  level  language  use  such  as  requirements  for 
additional  memory,  CPU  time  and  peripheral  devices.  Consideration 
should  be  given  to  developing  standard  means  whereby  these  benefits 
and  penalties  are  measured  (via  benchmark  programs,  relative  programm- 
ing cost  assessments,  etc.)  and  the  overall  benefit  (or  penalty) 
calculated.  Appropriate  standards  should  also  be  developed  for 
rating  computer  vendors'  compilers  for  generating  code  to  be  used  in 
real-time  simulation. 

2.  Should  all  simulator  systems  use  a common  data  base,  data  format 
and  preprocessor  for  the  preparation,  updating,  and  dissemination  of 
navigational  aid  data?  Updates  seem  too  often  initiated  as  a result 
of  instructor-pilot  or  trainee  reports  to  the  personnel  at  the  simu- 
lator site  that  the  simulator  no  longer  correctly  represents  the 
real  world  aids.  This  seems  to  be  a sufficiently  widespread  and 
frequently  occurring  simulator  data  change  activity  to  warrant  investi- 
gation of  means  of  expediting  updates  in  a uniform  manner. 

3.  Should  I/O  interface  equipment  and  interconnection  standards  be 
revised  or  augmented?  More  explicit  specification  requirements  were 
advocated  at  Altus  AFB  and  some  specific  suggestions  were  offered. 

A more  detailed  review  of  this  particular  area  seems  advisable. 
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